Creating a separate PR for incorrect use of ctx_split for bias tensors, following the suggestions by @slaren in in #8922 .

Please see this message for details.

Citing @slaren's comment

ctx_split only makes a difference when using tensor parallelism with -sm row, which is only supported on the CUDA backend when using multiple GPUs. When using -sm row, ctx_split splits the rows of the matrix between the available GPUs. This is only supported for matrix multiplication, so it should only be used with the matrix portion of linear/dense layers. The other cases are also wrong and should be corrected as well, but it doesn't need to be done here.

As far as I see, there are four such lines, which have been fixed in this PR.

Special thanks to @slaren !

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High